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The purpose of this study was to investigate the use of a critical thinking 
rubric as an assessment of reading achievement for students enrolled in a 
reading intervention course. A reading prompt and scoring rubric, based on 
Richard Paul and Linda Elder's critical thinking framework, were created 
to assess critical reading in an intervention course. The prompt and rubric 
were used throughout the semester to provide formative reading assessment. 
The scoring rubric, which is responsive to reading as a cognitive process, 
was also used for precourse and postcourse assessment to provide a unique 
measure of reading achievement that incorporates the university's critical 
thinking initiative. A repeated measures design was used to assess the reading 
achievement of 164 students on five different reading prompts over the course 
of a fall semester. Results showed significantly higher postassessment rubric 
scores (p <.001) and a significant change in rubric scores over time (p< .05). 

T 

* he search to measure reading 
achievement has an extensive history. Numerous standardized tests 
have been developed through the years to identify and measure specific 
reading skills, and research has been conducted to examine the validity 
of these tests. However, no single standardized reading test or informal 
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reading assessment is currently viewed to be the most effective means 
to assess college students' reading ability (Flippo & Schumm, 2009). A 
concern for designers of college reading instruction is that standardized 
tests used to assess students' reading achievement following college 
reading instruction may not measure a student's ability to read college- 
level reading assignments. Researchers who view reading as a strategic 
process have suggested that a student's mastery of reading subskills 
and improved performance on a standardized test may not demonstrate 
development of the college reading skills that contribute to academic 
success (Nist & Simpson, 2000; Simpson, Stahl, & Francis, 2004). 

Designers of college reading instruction also need reading tests to 
assess the development of reading skill over the course of a semester. To 
be effective, a reading course needs to provide instruction with authentic 
reading materials that enable students to reflect on and evaluate their 
reading, studying, and learning with the goal of controlling their own 
metacognitive processes (Simpson, Stahl, &" Francis, 2004). Reading 
courses need to explore innovative ways to help students improve their 
critical reading skills for success in college. Process-based approaches to 
reading and study strategies based on cognitive theory suggest that stu¬ 
dents need instruction in particular strategies to assist them to become 
successful students, including "how to select, transform, organize, elabo¬ 
rate, plan, monitor, and evaluate all critical thinking processes” (Nist & 
Simpson, 2000, p. 659), and the results of this instruction on learning 
should be assessed. Metacognitive assessments that promote reflection 
and critical thinking have been suggested as important to include in the 
college classroom and to consider for additional research (Flippo, Becker, 
& Wark, 2009). Reading assessments that help students determine an 
intentional purpose for reading and include specific criteria for evaluat¬ 
ing the reader's written responses may better indicate reading devel¬ 
opment than standardized tests (Farr, 2003). Individuals charged with 
choosing appropriate reading assessments should, therefore, consider the 
use of discipline-specific reading materials to provide authentic reading 
tasks. Discipline-specific reading materials from linked content courses 
may be appropriate to use to improve comprehension skills, even for 
developing college readers (Cox, Friesner, & Khayum, 2003). However, 
Flippo and Schumm (2009) indicated that little empirical evidence has 
been documented for the effectiveness of portfolios, authentic assess¬ 
ments, or performance-based assessments. 

This study sought to investigate how to assess the reading achievement 
of undergraduate students in a reading intervention course paired with 
general education courses. Reading achievement was measured by a 
required standardized reading test but additionally measured during the 
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semester and at the end of the semester by an alternative assessment 
technique, a written response evaluated by a scoring rubric. This study 
contributes empirical evidence that supports the need to continue inves¬ 
tigation of alternative assessment techniques in college reading courses. 

Background of the Study 

Required Statewide Assessment in Reading 

The National Governors Association Center for Best Practices and the 
Council of Chief State School Officers created a national initiative that 
resulted in the development of a set of educational standards referred to 
as the Common Core State Standards. These educational standards are 
intended to be a framework for preparing students for college and the 
workforce. These standards define the knowledge and skills that students 
should learn and develop in high school to be ready to succeed in entry- 
level, credit-bearing academic college courses and in workforce training 
programs. Each state that has adopted these standards is expected to 
create its own process for development and implementation. As a result, 
what students are expected to learn can vary widely from state to state. 
Kentucky is one of 45 states that have adopted these standards and is 
reforming curriculum in secondary schools to better align with entry- 
level curriculum in colleges and to assess college readiness (National 
Governors Association, 2010). 

Tb assist the implementation of the Common Core Standards, Kentucky 
state law defines a standard for college readiness in reading for public 
colleges and universities. All degree-seeking, first-year students with 
fewer than 24 credit hours admitted with ACT reading scores less than 
20 are required to participate in reading placement testing and course 
registration as determined by their reading placement results (Kentucky 
Developmental Education Thsk Force, 2007). To comply with state law, 
the university where this study was conducted selected the COMPASS 
Reading Test as a placement exam. An intervention course, Special 
Ibpics in Supplemented College Reading (GEN 105), was created in 2009. 
This first-year course is offered for one credit hour. The institution's 
centralized academic support unit instructs course sections of GEN 105 
each academic year. 

Students required to enroll in a course section of GEN 105 must concur¬ 
rently enroll in a selected section of a general education content course 
linked to the specific GEN 105 section. The content courses offered are 
in biology, psychology, and history. Students enrolled in GEN 105 are 
required to attend both the general education lecture course each week 
and two additional classroom sessions of GEN 105. In the GEN 105 class 
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sessions, students receive instruction in college reading, critical think¬ 
ing, and study strategies focused on the paired course. The textbook and 
supplemental readings assigned in each of the content courses serve as 
the basis for reading assignments and instruction in GEN 105. Grading 
for GEN 105 is pass or fail and appears as a separate course grade on 
the transcript. Reading competencies addressed in GEN 105 include the 
following: identify accurately the major purpose or concepts presented 
in a content reading; demonstrate a clear understanding of facts, data, or 
examples that support the purpose or concepts presented in a content 
reading; define precisely content-specific vocabulary explicit in a con¬ 
tent reading; develop well-reasoned and relevant conclusions based on 
content-specific information; and apply concepts from content reading 
broadly to other readings or non-reading contexts. 

Tb satisfy the university's need to participate in statewide assess¬ 
ment of students' reading achievement, the COMPASS Reading Tfest 
is administered at the beginning and end of the course. Placement 
(precourse) scores and end-of-semester (postcourse) scores are recorded 
and maintained as a standardized measure of the reading achievement 
of each student. These reading scores are used for institutional com¬ 
pliance reports. 

Scoring Rubric for Reading Assessment 

A different form of assessment, a reading prompt and scoring rubric, 
was constructed to assess students' critical reading in GEN 105, using the 
Paul and Elder framework of critical thinking applied to reading (Paul 
& Elder, 2009). Critical thinking is a key component of this institution's 
quality enhancement project for its next accreditation review. The uni¬ 
versity created the project and the task group to design instruction in 
critical thinking for faculty and staff and guide the curriculum revision 
process over the next ten years in all undergraduate units. The task group 
initially reviewed 13 different models of critical thinking. The Paul and 
Elder (2009) framework was unanimously adopted by the task group 
because of its comprehensiveness, applicability across disciplines, and 
high-quality resources. The designer of this reading intervention course 
chose to create a form of reading assessment that applied the university- 
selected critical thinking framework from a metacognitive perspective. 

Graduate student assistants are trained to instruct GEN 105. These 
instructors assign enrolled students to read and respond in writing to five 
discipline-specific readings selected from the content course textbook 
and supplemental course reading materials during the semester. The 
instructors assign a specific reading prompt to elicit students' written 
responses and evaluate each student's written response using a scoring 
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rubric based on the critical thinking framework. The reading prompt 
and scoring rubric focus students' attention on the development of their 
critical thinking skills. The scoring rubric is used as a formative assess¬ 
ment, providing detailed feedback from the instructor related to students’ 
levels of critical reading achievement at specified intervals during the 
semester (Stevens & Levi, 2005). These readings are assigned and scored 
about every two weeks throughout the semester, excluding the first and 
last weeks of the 14-week semester. Each time the instructor returns 
a scoring rubric with comments, students can evaluate their ability to 
comprehend discipline-specific reading material and note their ongoing 
improvement or lack of improvement toward the highest level of criti¬ 
cal reading achievement. The written responses and scoring rubrics are 
collected in an ungraded student portfolio to illustrate each student's 
efforts at the end of the semester (Stevens & Levi, 2005). The initial 
reading prompt and scoring rubric were reviewed and revised following 
the first year of the intervention. Data reported result from the revised 
prompt and rubric used in the second year of the intervention course. 

The scoring rubric also serves as a program assessment of students’ 
reading achievement at the end of the semester. During the first week 
of the semester, each student produces a written response to a reading 
prompt related to a discipline-specific reading excerpt from the required 
content course textbook. The student's written response is scored by the 
rubric and serves as a precourse assessment for students and a diagnostic 
test for instructors. During the last week of the semester, each student 
produces a written response to a reading prompt related to the same 
discipline-specific reading excerpt. The written response is then scored 
using the rubric and serves as a postcourse test. Comparisons of precourse 
and postcourse writings and of precourse and postcourse COMPASS Read¬ 
ing Test scores measure reading development. The COMPASS scores also 
provide standardized assessment data required to satisfy statewide assess¬ 
ment in reading, while the writing-task scores provide assessment data 
more representative of the reading and critical thinking tasks necessary 
for college courses ("Assessing Learning Outcomes," 2010). 

Paul and Elder Critical Thinking Framework 

The Paul and Elder (2009) framework uses a specific method of analy¬ 
sis and evaluation to instruct students in how to improve their critical 
thinking skills. This methodology provides a common framework identi¬ 
fying the Elements of Thought, which can be applied to a set of Universal 
Intellectual Standards with the goal of developing the Intellectual Traits. 
Paul and Elder illustrated the Elements of Thought using a circle diagram 
as shown in Figure 1 (Paul & Elder, 2009, p. 3). 
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Figure 1 

The Elements of Thought 
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Reprinted with permission from The Miniature Guide to Critical Thinking, by R. Paul and 
L. Elder, 2009, p. 3. Copyright 2009 by the Foundation for Critical Thinking. 
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The circle diagram presents components of all reasoning: purpose, 
question at issue, information, interpretation and inference, concepts, 
assumptions, and implications and consequences. Students can improve 
their critical thinking by analyzing their thinking using these elements. 
Students can create questions from each of the elements to better direct 
and focus their thinking, for example, "What is the purpose of what I am 
doing? What is the main question that I need to answer? What informa¬ 
tion do I need to answer that question?" Using this method, students 
learn to use a common vocabulary to focus their thinking, monitor their 
thinking, and correct faulty thinking. 

Students can apply the Universal Intellectual Standards to the Ele¬ 
ments of Thought to evaluate the quality of their critical thinking. This 
application is illustrated in Figure 2 (Paul & Elder, 2009, p. 19). When 
students learn to apply the standards of clarity, accuracy, relevance, 
logic, breadth, precision, significance, completeness, fairness, and depth 
to the Elements of Thought, they make judgments about the quality 
of their own thinking. According to Paul and Elder (2009), the goal is 
to develop particular Intellectual Traits: intellectual humility, intellec¬ 
tual autonomy, intellectual integrity, intellectual courage, intellectual 
perseverance, confidence in reasoning, intellectual empathy, and fair- 
mindedness. The development of these Intellectual Ttaits is essential 
to intellectual inquiry and warrants intentional instruction in critical 
thinking skills (Paul & Elder, 2009). 

Paul and Elder Critical Thinking Framework Applied to Reading 

Paul and Elder (2008) applied their framework of critical thinking to the 
reading process. Students who use good critical reading skills approach 
reading assignments based on the type of material read (Paul & Elder, 
2008). For example, a good critical reader reads a biology textbook differ¬ 
ently than a novel or magazine article. However, a good critical reader 
also recognizes the need to develop and apply general critical reading 
skills that are applicable to any type of reading because many different 
types of reading tasks maybe assigned in college. Paul and Elder (2008) 
suggested that students need to have a purpose for reading that considers 
the writer's purpose for writing. Students who are good critical readers read 
from paragraph to paragraph monitoring the author's thinking. During the 
reading process, students need to distinguish between their own thinking 
and the thinking of the author. For example, while students may read for 
their own purposes, using their own concepts, ideas, assumptions, infer¬ 
ences within their own personal points of view, they must also read "to 
recognize that embedded in the text is the author’s purpose, the author's 
question, assumptions, concepts and so forth" (Paul &" Elder, 2008, p. 8). 
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Figure 2 

The Paul-Elder Critical Thinking Framework 

Critical thinkers routinely apply intellectual standards to the 
elements of reasoning in order to develop intellectual traits. 
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Students can reconstruct the author's thinking by reflecting on their 
own personal experience of reading and recognize the interaction 
between reader and author. The Elements of Thought can serve as tools 
to help students reconstruct the author's thinking. Students who are good 
critical readers can reason through each of these elements for their own 
purposes, using their own concepts, ideas, assumptions, inferences, and 
personal points of view, but can also seek to recognize and understand 
the author's purpose, ideas, assumptions, inferences, and point of view. 
This method provides a means for helping students improve their criti¬ 
cal reading by reflecting on and evaluating their understanding, and by 
recognizing their own metacognitive processes (Paul & Elder, 2008). 
This critical reasoning can be demonstrated by asking students to write 
about what they understand from reading. 

Prompt for Reading Assessment 

The reading prompt is constructed to guide students through the 
reading and writing tasks. The reading prompt gives detailed directions 
to students and serves as the task description essential to evaluation 
by rubric (Stevens & Levi, 2005). Students are instructed to apply the 
prompt to the content reading from the linked content course (either 
an assigned section of the textbook or a supplemental reading directly 
related to a textbook topic). The reading prompt asks students to mark 
and annotate the text and then to write about their understanding of the 
content as presented by the author. Students are instructed to write an 
extended summary that includes very specific information: 

• the main purpose of the article (in 1 or 2 sentences); 

• the key concepts (facts and data) used to support the main 
purpose; 

• the article’s application to history, biology, or psychology; and 

• the article’s personal relevance to the student. 

The scoring rubric (Figure 3) is used to evaluate the student's writ¬ 
ten response. The rubric is based on a grid format with four parts: (a) 
the task description (reading prompt), (b) a scale indicating levels of 
achievement, (c) the categories or dimensions of the desired skills, and 
(d) descriptions for each level of performance (Stevens & Levi, 2005). 
The first vertical column in this scoring rubric provides the dimensions, 
which have been constructed using a combination of the Elements of 
Thought and the Universal Intellectual Standards from the Paul and 
Elder (2009) framework for critical thinking and the application of this 
method to reading. Each dimension represents a specific reading skill to 
be demonstrated and identifies each Intellectual Standard to be applied to 
the student's written performance reflecting the use of this reading skill. 


Figure 3 

Critical Thinking Rubric for Critical Reading 
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Student A's Written Response Assigned Low Ratings on the Critical Thinking Rubric 
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Student B's Written Response Assigned High Ratings on the Critical Thinking Rubric 
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The Intellectual Standards (accuracy, clarity, precision, depth, relevance, 
and logic) serve as the six dimensions of the rubric. The horizontal 
columns of the rubric create a rating scale to be applied to the differ¬ 
ing levels of student achievement. Evaluators use numbers 1 (lowest) 
through 4 (highest) to assess students' written performance. For each 
of the dimensions applying the six Intellectual Standards, categories 
provide a description for each potential level of performance. 

Training and Scoring 

Three graduate assistants were employed and trained in the summer 
before the intervention to teach the GEN 105 course sections offered in 
the fall semester. Each instructor was required to score sample student 
written responses in all three disciplines (biology, psychology, and his¬ 
tory) from the portfolios collected and retained from students enrolled 
in GEN 105 in the previous semester. In group training sessions led by 
professional trainers, the instructors completed scoring rubrics individu¬ 
ally and then compared and discussed ratings to establish benchmarks 
for each of the rubric scale levels for each of the three disciplines. In 
addition, each GEN 105 instructor was required to create a discipline- 
specific written response to serve as a model for each of the selected 
textbook or supplemental content readings used for the fall semester 
and for the reading excerpts or supplemental reading articles used for 
precourse and postcourse reading assessment in each of the disciplines 
to provide an additional means of improving the reliability of these 
instructors' evaluative ratings. The instructors were directed to use this 
same training procedure to introduce, explain, and model the reading 
prompt and rubric scoring during the first few weeks of instruction 
following precourse assessment administered in the first week of class. 

Tb illustrate this instructional procedure, two examples of students' 
written responses about an assigned biology textbook selection are pre¬ 
sented with accompanying scoring rubrics in Figures 4 and 5 (figures are 
found on the preceding pages). Student A illustrates a written response 
assigned low ratings (Figure 4). Student B illustrates a written response 
assigned high ratings (Figure 5). 
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Method 


Design of the Study 

The Paul and Elder (2009) critical thinking framework provides a 
theoretical connection between critical thinking and reading skill, but 
empirical evidence of the effectiveness of using this framework for 
instruction and assessment is needed. The purpose of this study was to 
investigate the use of a critical thinking rubric created using the Paul 
and Elder (2009) framework as an assessment of reading achievement 
for 164 students who had complete assessment data and were enrolled 
in a reading intervention course. The hypothesis for this study was that 
the use of this framework in the GEN 105 course would improve the 
reading achievement of students enrolled as measured by two differ¬ 
ent methods of assessment. This study asked three research questions: 
(1) Did students' scores indicate an increase in reading achievement 
as measured by the COMPASS Reading Test and the critical thinking 
rubric, an alternative measure of reading achievement? (2) What was 
the relationship between students' COMPASS Reading Tfest scores and 
critical thinking rubric scores? (3) What patterns and trends in reading 
achievement were suggested by the seven critical thinking rubric scores 
earned at spaced intervals during the semester? The study was approved 
by the university's Institutional Review Board as a retrospective review 
of existing data. 

Participant Characteristics 

The subjects in this study were 71 males (43.3%) and 93 females 
(56.7%) with a mean ACT of 17.5 (SD = 1.5) enrolled in seven sections of 
GEN 105 and paired content courses in biology, history, and psychology. 
Thble 1 summarizes the descriptive statistics for the subjects' COMPASS 
and critical thinking rubric scores. 


Table 1 

Dependent Variable Means, Standard Deviations, and Ranges 


Variable 

Mean 

SD 

Range 

preCOMPASS 

70.34 

10.67 

76 

postCOMPASS 

76.30 

11.46 

60 

preRubric 

16.69 

2.71 

12 

postRubric 

19.05 

2.92 

13 
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Results 

Reading Achievement Scores 

Tb answer the first research question and to determine the reading 
achievement indicated by students' mean score differences, paired- 
samples t-tests were computed. There was a statistically significant 
difference in the mean scores for the preCOMPASS Test and the post- 
COMPASS Tfest, t(141) = 5.587, p < .001. The postCOMPASS Tfest score 
mean (76.30) was significantly higher than the preCOMPASS Tfest score 
mean (70.34). Results also indicated a statistically significant difference 
in the mean scores for the precritical thinking rubric and the postcritical 
thinking rubric, 1(151) = 10.51, p < .001. The critical thinking postrubric 
score mean (19.05) was significantly higher than the critical thinking 
prerubric score mean (16.69). 

Relationship between COMPASS Reading Tfest and Critical 
Thinking Rubric Scores 

Tfe answer the second research question and to determine the strength 
of the relationship between the two reading achievement measures 
(COMPASS Reading Test scores and critical thinking rubric scores), a 
Pearson correlation was computed. No statistically significant relation¬ 
ship was found between the two measures (p > .05). This finding sug¬ 
gests that these two measures may be assessing reading achievement 
differently. 

Patterns and Trends in Reading Achievement over Time 

Tb answer the third research question regarding patterns and trends 
indicated by the seven critical thinking rubric scores that were earned 
at intervals during the semester, several repeated-measure ANOVAs 
(RM-ANOVA) were computed. One of the core underlying assumptions 
of the univariate RM-ANOVA is sphericity, meaning that the variances 
of the differences between data taken from the same participant are 
equal. Mauchly’s Test of Sphericity was computed and found to be 
highly significant, W = .76, % 2 (20) = 30.451, p = .063, suggesting that 
the observed matrix does have approximately equal variances and 
equal covariances, so no corrections were applied to the F-ratio compu¬ 
tations. Thble 2 summarizes the results of the RM-ANOVA analysis. 
A significant change in the students' critical thinking rubric scores did 
occur across time, F(6, 678) = 30.86, p < .05. 
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Table 2 

Repeated Measures Analysis of Variance for Critical Thinking Rubric 
Scores 


Sources 

SS 

df 

MS 

F 

P 

RM (Rubric over 
Time) 

936.208 

6 

156.035 

30.86 

.000 

Subjects 

3901.826 

113 

34.529 



Subjects x Rubric 
Over Time 

3428.078 

678 

5.056 



Total 

8266.112 

791 





In order to investigate these relationships further, follow-up contrasts 
compared all time points against the initial score (baseline). Post-hoc 
comparisons were performed using the Bonferroni adjustment for mul¬ 
tiple comparisons. As indicated in Thble 2, the impact of the repeated 
measures, the use of the rubric over time, is significant. 


Table 3 

Bonferroni Comparison for Critical Thinking Rubric Scores 


95% Cl 


Comparisons 

Mean Score 

Difference 

Std. Error 

Sig. 

Lower 

Bound 

Upper 

Bound 

Pre vs. Rubric 1 

.228 

.303 

1.000 

1.169 

.713 

Pre vs. Rubric 2 

1.202 

.323 

.006 

2.205 

.198 

Pre vs. Rubric 3 

2.632 

.322 

.000 

3.632 

1.631 

Pre vs. Rubric 4 

2.368 

.302 

.000 

3.308 

1.429 

Pre vs. Rubric 5 

2.772 

.304 

.000 

3.718 

1.826 

Pre vs. Post 

2.377 

.255 

.000 

3.169 

1.585 









50 


Journal of College Reading and Learning, 43(1), Fall 2012 


The Bonferroni comparisons can be seen in Table 3; in this section, 
the mean scores that underpin the data in that table are discussed. The 
critical thinking rubric scores did not increase significantly from a mean 
of 16.78 ( SD = 2.77) for the precourse rubric score to a mean of 17.00 
(SD = 2.9) for rubric 1 (Rl). However, significant improvement was found 
at the following points across time: rubric 2 (R2) (M = 17.98, SD = 3.26); 
rubric 3 (R3) (M = 19.41, SD = 3.39); rubric 4 (R4) (M = 19.14, SD = 3.17); 
rubric 5 (R5) (M = 19.55, SD = 2.96); postcourse rubric score (M = 19.15, 
SD = 2.80). Figure 6 shows the marginal means for the critical thinking 
rubric scores at each point of measurement. 

To examine the critical thinking rubric scores across time for each of 
the different paired courses in biology, psychology, and history, a RM- 
ANOVA was again conducted. Figure 7 shows the marginal means for each 
discipline at each point of observation. Students did show statistically 
significant improvements as measured by the means of the precourse, 
postcourse, and rubric scores 1-5 for each discipline; however, for each 
discipline, improvements occurred among different readings and at dif¬ 
ferent data points in the semester. 

Biology trend. Mauchly’s test indicated that the assumption of sphericity 
had been violated (W = .15, df = 20, p < .05) for the biology GEN 105 sec¬ 
tions; therefore, degrees of freedom were corrected using Greenhouse-Geisser 
estimates of sphericity (e = .565). The results indicated a significant effect on 
the rubric scores across time in the biology GEN 105 course sections, F( 3.39, 
94.95) = 4.92, p < .05. The critical thinking rubric scores did not increase 
significantly from a mean of 14.41 (SD = 2.69) for the precourse rubric 
score to a mean of 15.51 (SD = 2.5, p = 1.000) for rubric 1 (Rl). In addition, 
no statistically significant improvement was found across time for rubric 2 
(M = 16.14, SD = 2.34, p = .085) nor for rubric 3 (M = 16.72, SD = 2.34, 
p = .482). However, significant improvement was found across time for 
rubric 4 (M = 16.89, SD = 2.52, p < .05), rubric 5 (M = 17.17, SD = 2.59, 
p < .05), and the postcourse rubric scores (M = 17.00, SD = 2.26, p < .05). 

History trend. Mauchly’s test indicated that the assumption of sphericity 
had not been violated (W = .506, df = 20, p = .116) for the history 
GEN 105 course sections; therefore, no corrections were applied to the 
F-ratio computations. The results indicated a significant effect on the 
critical thinking rubric scores across time in the history GEN 105 course 
sections, F( 6, 258) = 18.02, p < .05. The critical thinking rubric scores did 
not increase significantly from a mean of 16.61 (SD = 2.27) for the pre¬ 
course rubric score to a mean of 17.52 (SD = 3.25, p = 1.000) for rubric 1 
(Rl). However, significant improvement was found across time for rubric 2 
(M= 18.61, SD = 3.84, p< .05), rubric3(M = 19.61, SD = 2.44,p < .05),mbric4 
(M = 19.25, SD = 3.32, p < .05), rubric 5 (M = 20.84, SD = 2.50, p < .05), 
and the postcourse rubric scores (M = 19.25, SD = 2.87, p < .05). 


Figure 6 

Estimated Marginal Means of Rubric Scores Over Time for All GEN 105 Courses 

20.00 
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Figure 7 

Estimated Marginal Means of Rubric Scores Over Time for Each GEN 105 Discipline 
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Psychology trend. Mauchly’s test indicated that the assumption of 
sphericity had not been violated (W = .505, df = 20, p = .173) for the 
psychology GEN 105 course sections; therefore, no corrections were 
applied to the F-ratio computations. The results indicated a significant 
effect on the critical thinking rubric scores across time in the psychol¬ 
ogy GEN 105 course sections, F(6, 240) = 16.88, p < .05. The critical 
thinking rubric scores did not increase significantly from a mean of 
18.63 (SD = 1.86) for the precourse rubric score to a mean of 17.51 
(SD = 2.41, p = .369) for rubric 1 (Rl), nor was statistically significant 
improvement found across time for rubric 2 (M = 18.60, SD = 2.63, p 
= 1.000). However, significant improvement was found across time for 
rubric 3 (M = 21.09, SD = 1.97, p < .05) and rubric 4 (M = 20.63, SD 
= 2.46, p < .05). No statistical improvement of scores was found for 
rubric 5 (M = 19.85, SD = 2.70, p = .274), although the precourse to 
postcourse rubric scores indicated significant differences, (M = 20.58, 
SD = 2.03, p < .05). 

Examining the critical thinking standards over time, lb examine 
the relationships of the individual dimensions comprising each critical 
thinking rubric score further, an analysis was conducted for the fol¬ 
lowing: accuracy, clarity, precision, depth, relevance, and logic across 
the semester time span. Figure 8 shows the marginal means for each 
dimension at each time of assessment. 

No statistically significant difference was found for any of the mean 
scores for each dimension of the total precourse rubric score and the rubric 
1 score (p = 1.00). However, statistically significant differences were found 
for some mean scores for different dimensions and for different readings 
across the remaining time of the semester. Students' scores on accuracy, 
clarity, and precision showed significant mean differences when compar¬ 
ing the precourse rubric score with scores earned on rubrics 2, 3, 4, 5, and 
the postcourse rubric (p < .05), and students’ scores on clarity showed 
significant mean differences when comparing the precourse rubric score 
with scores earned on rubrics 3, 4, 5, and the postcourse rubric (p < .05). 
Students' scores on precision showed significant mean differences when 
comparing the precourse rubric score with scores earned on rubrics 2, 3, 
4, 5, and the postcourse rubric (p < .05). Students’ scores on depth showed 
significant mean differences when comparing the precourse rubric score 
with scores earned on rubrics 3, 4, 5, and the postcourse rubric (p < .05). 
The dimensions of relevance and depth showed the least improvement 
over time. Students’ scores on relevance showed significant mean differ¬ 
ences only when comparing the precourse rubric to rubrics 3, 4, 5 (p < 
.05), and students' scores on logic showed a significant mean difference 
only between the precourse rubric score and rubric 4 (p < .05). 


gure 8. 

arginal Means for Rubric Dimensions Over Time 

3.70 
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Discussion 

The significant difference found between the students' mean place¬ 
ment (precourse) and end of semester (postcourse) COMPASS Reading 
Tfest scores and the significant difference found between the mean scores 
for precourse and postcourse rubric scores are encouraging for these 
designers of college reading instruction. These findings support that 
students enrolled in GEN 105 improved performance on two different 
measures of reading achievement following the instruction received in 
the GEN 105 course sections across all three disciplines. The lack of a 
statistically significant relationship between the students' rubric scores 
and COMPASS Reading Test scores may indicate that the standardized 
reading test and the written summary are measuring different subskills 
related to reading achievement and cognitive reasoning. The products 
measured (correct multiple-choice responses vs. a written summary) 
are very different methods of assessment. 

For all three disciplines, students did show statistically significant 
improvements in rubric scores. However, rubric scores did not increase 
significantly from the precourse score to the completion of rubric 1. 
This finding maybe the result of the GEN 105 curriculum as structured 
during the fall semester. Most instruction in the Paul and Elder (2009) 
framework and in-class explanation of the rubric’s dimensions occurred 
between rubric 2 and rubric 3. The late timing of this instruction may 
have resulted in less improvement in the critical thinking rubric scores 
very early in the semester. In addition, differences in the assigned course 
readings among the three disciplines may also have had a significant 
effect on this study’s results. A review of the number of pages assigned 
in the textbook and the number of outside readings assigned found the 
history course to be the most reading-intensive of these three courses, 
psychology to be the second most reading-intensive course, and biology 
to be the least reading-intensive course. Differences in the quantity of 
reading assigned do correspond to this study’s results, which indicate 
statistically significant improvement in critical thinking rubric scores 
earned following rubrics 2, 3, 4, and 5 in history. Psychology showed 
similar results only after rubrics 3 and 4, while biology showed similar 
results only after rubrics 4 and 5. Lastly, biology and history results 
show (in Figure 7) a consistent positive slope for improving rubric scores 
across time to rubric 3, but this same result was not found for psychol¬ 
ogy. Scheduling the due date for rubric 3 in GEN 105 on the same day 
as the first psychology exam may have contributed to this result by 
decreasing student attention to the critical reading task. 

Additional training has already been implemented for the graduate 
student assistants selected to instruct course sections of GEN 105 for 
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future semesters. This training includes generating an inter- and intra¬ 
reader reliability score during a longer and more extensive training 
process in the Paul and Elder (2009) critical thinking framework to 
strengthen the consistency of rubric scoring among these instructors. 
Group scoring sessions were also implemented following the scoring 
of rubric 2 across all three disciplines as an additional training experi¬ 
ence to improve scoring reliability and validity. Tfextbook excerpts and 
articles selected and assigned as the critical readings are scrutinized 
carefully for readability, length, and relation to topic. Attention was 
given to providing similar although supplemental reading experiences 
across the three disciplines. 

Limitations of the Study 

One limitation of this study is the lack of a control group; it is not pos¬ 
sible to know the strength of factors besides reading instruction on read¬ 
ing achievement gains. The instruction received and the college-level 
reading assignments completed concurrently in the content courses are 
assumed to be contributing factors to students' improved reading achieve¬ 
ment, and maturation may be another factor; therefore the intentional 
instruction of the Paul and Elder (2009) critical thinking framework to 
improve reading skills was not the sole contribution to students' read¬ 
ing achievement. In addition, this study is limited because it presents 
only one semester's data. Additional semesters of data are needed for 
additional analysis. A larger number of scores creating a larger data set 
over time may produce different results. 

Future Research 

Additional research is needed to explore the use of the critical thinking 
rubric to assess college reading achievement. If students are expected 
to read complex and diverse content-related text critically for success in 
college, then assessment of college programs and courses that address 
college readiness in reading must continue to evolve and to provide 
data that best demonstrate readiness for entry-level college courses. 
Accountability for exploring multiple measures of reading achievement 
that more accurately reflect the challenges of the college classroom may 
best be assigned to instructors who create curriculum and prepare stu¬ 
dents for college readiness. College administrators who are accountable 
for measures of college readiness for reading for state-wide reporting 
will want to encourage this exploration by instructors of developmental 
or intervention courses. At this institution, additional procedures are 
planned to further explore and improve the use of the critical thinking 
rubric for reading assessment in GEN 105. Future research is planned 
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to compare data from multiple years to review the possible effects 
from additional instructor training and from improved focus on select¬ 
ing readings for the content areas and may provide more evidence for 
better understanding the learning outcomes of using this method of 
assessment for college reading. 
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